Read My Lips: Towards Use of the Microsoft Kinect as a Visual-Only Automatic Speech Recognizer

Consumer devices used in the home are capable of collecting ever more information from users, including audio and video. The Microsoft Kinect is particularly well-designed for tracking user speech and motion. In this paper, we explore the ability of current models of the Kinect to support use as an automatic speech recognizer (ASR). Lip reading is known to be difficult due to the many possible lip motions. Our goals were to quantify lip movement while observing the correlation with recognized words. Our preliminary results show that word recognition through the audio interface and with use of the Microsoft Speech API can provide upwards of 90% accuracy over a corpus of words, and that the visual acuity of the Kinect is such that we can capture a total of 22 data points representing the lip model through the Face Tracking API at a high resolution. Based on these results and that of recent work, we forecast that the Kinect has the ability to act as an ASR and that words can potentially be reconstructed through the observation of lip movement without the presence of sound. Such an ability for household devices to observe and parse communication presents a new set of privacy challenges within the home.